8506 


PATENT APPLICATION 


Certificate of "Express Mail " 


Filing Number 
Date of Deposit 


ET 500 734 359 US 
May 30, 2001 


I hereby certify that the enclosed patent application and filing fee are being deposited with 
the United States Postal Service "Express Mail Post Office to Addressee" service under 37 CFR 1.10 
on the date indicated above and is addressed to: 

Box Patent Application 

Honorable Assistant Commissioner of Patents 

Washington, D.C. 20231 



THIS IS A REQUEST FOR FILING A PATENT APPLICATION 
Of 

IZRAIL S. GORIAN 

233 Warren Street 
Watertown, MA 02472 

JAY E. THORNTON 

56 Lincoln Street 
Watertown, MA 02472 

and 

RICHARD A. PINEAU 

395 Chestnut Street 
No. Andover, MA 01845 

For 

RENDERING IMAGES UTILIZING 
ADAPTIVE ERROR DIFFUSION 


Polaroid Corporation 

Patent Department 

784 Memorial Drive 

Cambridge, Massachusetts 02139 

Tel: (781)386-6063 

Fax: (781) 386-6435 


Respectfully submitted, 



8506 


PATENT APPLICATION 


RENDERING IMAGES UTILIZING 
ADAPTIVE ERROR DIFFUSION 

BACKGROUND OF THE INVENTION 

Field Of The Invention 

The present invention relates to the rendering of digital image data, and in 
particular, to the binary or multilevel representation of images for printing or display 
purposes 

Background Description 

Since images constitute an effective means of communicating information, 
displaying images should be as convenient as displaying text. However, many display 
devices, such as laser and ink jet printers, print only in a binary fashion. Furthermore, 
some image format standards only allow binary images. For example, the WAP 1.1 
(Wireless Application Protocol) protocol specification allows only for one graphic 
format, WBMP, a one (1) bit version of the BMP (bitmap) format. Besides allowing 
only binary images, some image format standards and some displays only allow 
images of a limited number of pixels. In the WAP 1.1 standard, a WBMP image 
should not be larger than 150 x 150 pixels. Some WAP devices have screens that are 
very limited in terms of the number of pixels. For example, one WAP device has a 
screen that is 96 pixels wide by 65 pixels high. In order to render a digitized 
continuous tone input image using a binary output device, the image has to be 
converted to a binary image. 

The process of converting a digitized continuous tone input image to a 


l 


8506 


PATENT APPLICATION 


binary image so that the binary image appears to be a continuous tone image is known 
as digital halftoning. 

In one type of digital halftoning processes, ordered dither digital halftoning, 
the input digitized continuous tone image is compared, on a pixel by pixel basis, to a 
threshold taken from a threshold array. Many ordered dither digital halftoning 
methods suffer from low frequency artifacts. Because the human vision system has 
greater sensitivity at low frequencies (less than 12 cycles/degree), such low frequency 
artifacts are very noticeable. 

The visibility of low frequency artifacts in ordered dither digital halftoning 
methods has led to the development of methods producing binary images with a 
power spectrum having mostly higher frequency content, the so called "blue noise 
methods". 

The most frequently used "blue noise method" is the error diffusion method. 
In an error diffusion halftoning system, an input digital image I n (the digitized 
continuous tone input image) is introduced into the system on a pixel by pixel basis, 
where n represents the input image pixel number. Each input pixel has its 
corresponding error value E„.i, where E n .i is the error value of the previous pixel (n- 
1), added to the input value I n at a summing node, resulting in modified image data. 
The modified image data, the sum of the input value and the error value of the 
previous pixel (I n + E n _i), is passed to a threshold comparator. The modified image 
data is compared to the constant threshold value T. 0 , to determine the appropriate 
output level O n . Once the output level O n is determined, it is subtracted from the 
modified image value to produce the input to an error filter. The error filter allocates 
its input, I n - O n , to subsequent pixels based upon an appropriate weighting scheme. 
Various weighting techniques may be used generate the error level E. n for the 
subsequent input pixel. The cyclical processing of pixels is continued until the end of 
the input data is reached. (For a more complete description of error diffusion see, for 
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example, "Digital Halftoning", by Robert Ulichney, MIT Press, Cambridge, Mass. 
and London, England, 1990, pp. 239-319). 

Although the error diffusion method presents an improvement over many 
ordered dither methods, artifacts are still present. There is an inherent edge 
enhancement in the error diffusion method. Other known artifacts produced by the 
error diffusion method include artifacts called "worms" and "snowplowing" which 
degrade image quality. 

In U.S. Patent No. 5,045,952, Eschbach disclosed selectively modifying the 
threshold level on a pixel by pixel basis in order to increase or decrease the edge 
enhancement of the output digital image. The improvements disclosed by Eschbach 
do not allow the control of the edge enhancement by controlling the high frequency 
portion of the error. Also, the improvements disclosed by Eschbach do not introduce 
parameters that can be selected to produce the image of the highest perceptual quality 
at a specific output device. 

In U.S. Patent No. 5,757,976, Shu disclosed utilizing a set of error filters 
having different sizes for diffusing the input of the error filter among neighboring 
pixels in predetermined tonal areas of an image and adding "noise" to the threshold in 
order to achieve a smooth halftone image quality. The improvements disclosed by 
Shu do not introduce parameters that can be selected to produce the image of the 
highest perceptual quality at a specific output device. 

SUMMARY OF THE INVENTION 

It is the primary object of this invention to provide a method for generating 
a halftone image from a digitized continuous tone input image that provides 
adjustment of the local contrast of the resulting halftone image, minimizes artifacts 
and is easily implemented. 
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It is also an object of this invention to provide a method for generating a 
halftone image with parameters that can be selected to produce the image of highest 
quality at a specific output device. 

To achieve the objects of this invention, one aspect of this invention includes 
an adaptive halftoning method where the difference between a digital image and a 
filtered digital image is introduced into the system on a pixel by pixel basis; each 
input difference pixel having a corresponding error value, generated from the previous 
pixels, added to the input value at a summing node, resulting in modified image 
difference data; the modified image difference data being passed to a threshold 
comparator where the modified image difference data is compared to a threshold 
value, the threshold value varying according to the properties of the digital image, to 
determine the appropriate output level; the output level is subtracted from the 
modified image difference value to produce the input to an error filter; the output of 
the error filter is multiplied by a adaptation coefficient, where the adaptation 
coefficient varies according to the properties of the digital image, to generate the error 
level for the subsequent input pixel; and, the cyclical processing of pixels is continued 
until the end of the input data is reached. 

In another aspect of this invention, in the method described above, a 
histogram modification is performed on the image, and the difference between the 
histogram modified digital image and the filtered digital image is introduced into the 
system on a pixel by pixel basis. 

In still another aspect of this invention, in the method described above, the 
histogram modification is performed on the difference between the digital image and 
the filtered digital image and the histogram modified difference is introduced into the 
system on a pixel by pixel basis. 

In a further aspect of this invention, in the method described above, the 
selectively changing of the adaptation coefficient comprises dividing the difference 
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between the value at the pixel and the filtered value at the pixel by the filtered value at 
the pixel, multiplying the absolute value of the result of the division by a first 
parameter, and adding a second parameter to the result of the multiplication, thereby 
obtaining the coefficient. 

In still another aspect of this invention, in the method described above, the 
threshold calculation comprises multiplying the filtered value at the pixel by a third 
parameter. 

In still another aspect of this invention, in the method described above and 
including the adaptation coefficient and threshold calculated as in the two preceding 
paragraphs, where the filter is a filter of finite extent, the extent of the filter, the first, 
second parameters and third parameters are selected to produce the image of the 
highest perceptual quality at a specific output device. 

The methods, systems and computer readable code of this invention can be 
used to generate halftone images in order to obtain images of the highest perceptual 
quality when rendered on displays and printers. The methods, systems and computer 
readable code of this invention can also be used to for the design of computer 
generated holograms and for the encoding of the continuous tone input data. 

DESCRIPTION OF THE DRAWINGS 

The novel features that are considered characteristic of the invention are set 
forth with particularity in the appended claims. The invention itself, however, both as 
to its organization and its method of operation, together with other objects and 
advantages thereof will be best understood from the following description of the 
illustrated embodiment when read in connection with the accompanying drawings 
wherein: 

Fig. la depicts a block diagram of selected components of an embodiment 
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of a system, of this invention for generating a halftone image from a digitized 
continuous tone input image, where the histogram modification block is included after 
the summing node; and, 

Fig. lb depicts a block diagram of selected components of an embodiment 
of a system of this invention for generating a halftone image from a digitized 
continuous tone input image, where the histogram modification block is included 
before the summing node; and, 

Fig. lc depicts a block diagram of selected components of an embodiment 
of a system of this invention for generating a halftone image from a digitized 
continuous tone input image, where the adaptation coefficient multiplies the input to 
the error filter block; and 

Fig. 2 depicts a block diagram of selected components of another 
embodiment of the system of this invention for generating a halftone image from a 
digitized continuous tone input image; and 

Fig. 2a depicts a block diagram of selected components of another 
embodiment of the system of this invention for generating a halftone image from a 
digitized continuous tone input image, where the adaptation coefficient multiplies the 
input to the error filter block. 

DETAILED DESCRIPTION 

A method and system, for generating a halftone image from a digitized 
continuous tone input image, that provide adjustment of the local contrast of the 
resulting halftone image, minimizes artifacts, are easily implemented and contain 
parameters that can be selected on the basis of device characteristics like brightness, 
dynamic range, and pixel count, to produce the image of highest perceptual quality at 
a specific output device are disclosed. 

6 


8506 


PATENT APPLICATION 


A block diagram of selected components of an embodiment of a system of 
this invention for generating a halftone image from a digitized continuous tone input 
image (also referred to as a digital image) is shown in Fig. la. Referring to Fig. la, 
image input block 10 introduces an input digital image I n into the system on a pixel 
by pixel basis, where n represents the input image pixel number. The input image is 
also provided to the filtering block 20. The output of filtering block 20 has the form 

Av n =h(..,I k ,..,I. n ,...) 0) 


where h is a functional form spanning a number of pixels. It should be apparent that 
the input digital image 10 can be a two dimensional array of pixel values and that the 
array can be represented as a linear array by using such approaches as raster 
representations or serpentine representation. For a two dimensional array of pixel 
values, the filter 20 will also be a two dimensional array of filter coefficients and can 
also be represented as a linear array. The functional forms will be shown in the one 
dimensional form for ease of interpretation. 

In one embodiment: the output of the filtering block 20 has the form 


n+N 

Av n ={2 n . N Ij}/(2N+l) (2) 
If the filtering block 20 comprises a linear filter, Av „ will be given by a sum of 
terms, each term comprising the product of an input image pixel value multiplied by 
a filter coefficient. 

It should be apparent that special consideration has to be given to the pixels 
at the boundaries of the image. For example, the calculations can be started N pixels 
from the boundary in equation (2). In that case the calculated and halftone image are 
smaller than the input image. In another case, the image is continued at the 
boundaries, the continuation pixels having the same value as the boundary pixel. It 
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should be apparent that other methods of taking into account the effect of the 
boundaries can be used. 

The output of the filtering block 20, Av n , is subtracted from the input 
digital image I. n at node 25, resulting in a difference value, D n . In the embodiment 
in which histogram modification is not included, D n is the input to a summing node 
70. At the summing node 70, a corresponding error value E n -i, where E„-i is the 
error value accumulated from the previous pixels , is added to the input value D n 
resulting in a modified image datum. The modified image data, D n + E n _i, is 
compared to the output of the threshold calculation block 30 in the threshold 
comparison block 40 to produce the halftoning output, O n . (In the case of a binary 
output device, if the modified image datum is above the threshold, the output level is 
the white level. Otherwise, the output level is the black level.) Once the output level 
O n is determined, it is subtracted from the modified image value to produce the input 
to an error filter block 50. The error filter block 50 allocates its input, D n + E n -i- O n , 
to subsequent pixels based upon an appropriate weighting scheme. The weighted 
contributions of the error filter block 50 input are stored and all the contributions to 
the next input pixel are summed to produce the output of the error filter block 50, the 
error value. The output of the error filter block 50, the error value, is multiplied by 
the adaptation coefficient in block 60 to generate the error level E. n for the 
subsequent input pixel. The cyclical processing of pixels, as further described below, 
is continued until the end of the input data is reached. 

Referring again to Fig. 1, the input image is also provided to the threshold 
calculation block 30. The output of the threshold calculation block 30 has the form 

t(..,I k ,..,I. n ,...) (3) 

where t is a functional form spanning a number of pixels. The form in equation (3) 
allows the varying of the threshold according to properties of the digital image. 
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In one embodiment, 


n+N 


t ( .., I. n ,...) = C 0 { S „. N Ij }/ (2N+1) 


(4) 


In another embodiment, the output of the threshold calculation block is a linear 
combination of terms, each term comprising the product of an input image pixel value 
multiplied by a coefficient. It should be apparent that this embodiment can also be 
expressed as a function times a parameter. 

The output of the threshold calculation block 30 is the threshold. 

The first pixel value to be processed, Io, produces a difference value Do 
from summing node 25 and produces a value of Do out of summing node 70 (since 
E_i is equal to 0). Do is then compared to the threshold producing an output of O 0 . 
At summing node 45, Oo is subtracted from Do to produce the input to the error 
filter 50. The error filter 50 allocates its input, D 0 - O 0 , to subsequent pixels based 
upon an appropriate weighting scheme which determines how much the current input 
contributes to each subsequent pixel. Various weighting techniques may be used (see, 
for example, "Digital Halftoning" by Robert Ulichney, MIT Press, Cambridge, Mass. 
and London, England, 1990, pp. 239-319). The output of error filter 50 is multiplied 
by a adaptation coefficient 60. The adaptation coefficient 60 is the output of the 
coefficient calculation block 80. In one embodiment, the output of the coefficient 
calculation block 80 has the form 


where f and g are functional forms spanning a number of pixels. The form of 
Equation (5) allows the selective changing, of the coefficient according to the local 
properties of the digital image. Ci and C 2 and the parameter in the threshold 
expression can be selected to produce the image of highest perceptual quality at a 


d + C 2 abs{ f ( ..,I k , .., I. n ,...)/ g ( -,1k, !•„,.. .)} 


(5) 
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specific output device. 

In another embodiment, the output of the coefficient calculation block 80 
has the form 

n+N n+N 

d + C 2 {abs((I. n - ({ I n . N Ij }/ (2N+1)) )/ ({ S „-n Ij }/ (2N+1))))} (6) 

The input of error filter block 50 is multiplied by weighting coefficients 
and stored. All the contributions from the stored weighted values to the next pixel are 
summed to produce the out put of the error filter block 50. The output of the error 
filter block 50 is multiplied by the adaptation coefficient 60. The delay block 65 
stores the result of the product of the adaptation coefficient 60 and the output of the 
error filter block 50. (In one embodiment, the Floyd-Steinberg filter, the input to the 
error filter is distributed according to the filter weights to the next pixel in the 
processing line and to neighboring pixels in the following line.) The output of delay 
block 65 is E n -i and is delayed by one pixel. (When the first pixel is processed, the 
output of the delay, Eo , is added to the subsequent difference, Di .) 

It should be apparent that the sequence order of error filter block 50 and the 
adaptation coefficient block 60 can be interchanged with similar results. In the 
embodiment in which the adaptation coefficient 60 multiplies the difference between 
the modified image datum and the output level, shown in Fig. lc, the delay block 65 
stores the output of the error filter block. 

When the next pixel, Ii ; is introduced into the system from the image input 
block 10, it produces a difference value Di from summing node 25 and produce a 
value of (Di +E 0 ) out of summing node 70. 

The above steps repeat for each subsequent pixel in the digital image thereby 
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producing a halftone image, the sequence O 0 , Oi, , 0„. The modification of 

the threshold level and the adaptation coefficient allows control of the amount of edge 
enhancement and provides the opportunity to reduce artifacts. 

In the embodiment in which histogram modification is included after the 
summing node 25, D n is the input to the histogram modification block 75 and the 
output of the histogram modification block 75 is the input to the summing node 70. 
The above description follows if D„ is replaced by the output of the histogram 
modification block 75. It should be apparent that histogram modification operates on 
the entire difference image. (Histogram modification is well known to those skilled in 
the art. For a discussion of histogram modification, see, for example, Digital Image 
Processing, by William K. Pratt, John Wiley and Sons, 1978, ISBN 0-471-01888-0, 
pp. 311-318. For a discussion of histogram equalization, a form of histogram 
modification, see, for example, Digital Image Processing, by R. C. Gonzalez and P. 
Wintz, Addison- Wesley Publishing Co., 1977, ISBN 0-201-02596-3, pp. 1 19-126.) 

In the embodiment in which histogram modification is included after the 
image input block 10, D„ is the difference between the output of the histogram 
modification block 75 (Fig. lb) and the filtered image. The above description 
follows if I n is replaced by the output of the histogram modification block. 

The method described above produces improvements of the error diffusion 
method by utilizing the difference between the digital image and the filtered digital 
image as input into the system instead of the digital image, by multiplying the .the 
output of the error filter by the adaptation coefficient, where the adaptation coefficient 
varies according to the properties of the digital image, and by using a threshold value 
that varies according to the properties of the digital image to determine the 
appropriate output level. 

Sample Embodiment 

In a specific embodiment, shown in Fig. 2, the output of the filtering block 
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20, Av n , is given by Equation (2). The threshold calculation 30 is a function of the 
output of the filtering block 20 and is given by 

t (.., I k , ..,I. n ,-..) = Co Av„ (7) 

which is the same function as in Equation 4 when the output of the filtering block 20, 
Av n , is given by Equation (2). The output of the coefficient calculation block 80 
depends on the output of the filtering block 20, Av n , and the difference D„ and is 

given by 

Ci + C 2 {abs( (D„ - Av n )/Av„ ) } (8) 

When the output of the filtering block 20, Av n , is given by Equation (2), Equation 
(8) is the same as Equation (6). 

Histogram equalization is included after the summing node 25. The 
processing of the input image pixels 10 occurs as described in the preceding section. 

The value of N in Equation (2) (the extent of the filter), C 0 , Cj, and C 2 
(first, second parameters and third parameters) can be selected to produce the image 
of highest perceptual quality at a specific output device. For a WBMP image on a 
specific monochrome mobile phone display, utilizing a Floyd-Steinberg error filter, 
the following parameters yield images of high perceptual quality: 

N =7, 

Co = -20, 

d = 0.05, and 

C 2 =l. 


In another embodiment, shown in Fig. 2a, the sequence order of error filter block 50 
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and the adaptation coefficient block 60 are interchanged. In the embodiment of Fig. 
2a, in which the adaptation coefficient 60 multiplies the difference between the 
modified image datum and the output level, the delay block 65 stores the output of the 
error filter block. 

The embodiments described herein can also be expanded to include 
composite images, such as color images, where each color component might be 
treated individually by the algorithm. In the case of color input images, the value of 
N in Equation (2) (the extent of the filter), Co, Ci, and C2 (first, second parameters 
and third parameters) can be selected to control the color difference at a color 
transition while minimizing any effects on the brightness at that location. Other 
possible applications of these embodiments include the design of computer generated 
holograms and the encoding of the continuous tone input data. 

Although the embodiments described herein are most easily understood for 
binary output devices, the embodiments described herein can also be expanded to 
include rendering an output image when the number of gray levels in the image 
exceeds that of obtainable in the rendering device. It should be apparent how to 
expand the embodiments described herein to M-ary displays or M-ary rendering 
devices (see, for example, "Digital Halftoning" by Robert Ulichney, MIT Press, 
Cambridge, Mass., and London, England, 1990, p. 341). 

It should be appreciated that the various embodiments described above are 
provided merely for purposes of example and do not constitute limitations of the 
present invention. Rather, various other embodiments are also within the scope of the 
claims, such as the following. The filter 20 can be selected to impart the desired 
functional behavior of the difference. The filter 20 can, for example, be a DC 
preserving filter. The threshold 40 and the adaptation coefficient 60 can also be 
selected to impart the desired characteristics of the image. 

It should be apparent that Equations (4) and (5) are exemplary forms of 
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functional expressions with parameters that can be adjusted. Functional expressions 
for the threshold and the adaptation coefficient ,where the expressions include 
parameters that can be adjusted , will satisfy the object of this invention. 

In general, the techniques described above may be implemented, for 
example, in hardware, software, firmware, or any combination thereof. The 
techniques described above may be implemented in one or more computer programs 
executing on a programmable computer including a processor, a storage medium 
readable by the processor (including, for example, volatile and non-volatile memory 
and/or storage elements), at least one input device, and at least one output device. 
Program code may be applied to data entered using the input device to perform the 
functions described and to generate output information. The output information may 
be applied to one or more output devices. 

Elements and components described herein may be further divided into 
additional components or joined together to form fewer components for performing 
the same functions. 

Each computer program within the scope of the claims below may be 
implemented in any programming language, such as assembly language, machine 
language, a high-level procedural programming language, or an object-oriented 
programming language. The programming language may be a compiled or interpreted 
programming language. Each computer program may be implemented in a computer 
program product tangibly embodied in a machine-readable storage device for 
execution by a computer processor. Method steps of the invention may be performed 
by a computer processor executing a program tangibly embodied on a computer- 
readable medium to perform functions of the invention by operating on input and 
generating output. 

The generation of the halftone image can occur at a location remote from the 
rendering printer or display. The operations performed in software utilize instructions 
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("code") that are stored in computer-readable media and store results and intermediate 
steps in computer-readable media. 

Common forms of computer-readable media include, for example, a floppy 
disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a 
CDROM, any other optical medium, punch cards, paper tape, any other physical 
medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH- 
EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, 
or any other medium from which a computer can read. Electrical, electromagnetic or 
optical signals that carry digital data streams representing various types of 
information are exemplary forms of carrier waves transporting the information. 

Other embodiments of the invention, including combinations, additions, 
variations and other modifications of the disclosed embodiments will be obvious to 
those skilled in the art and are within the scope of the following claims. 
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